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1.  Introduction 


Soldiers  in  theater  taking  fire  from  hostile  forces  often  have  limited  situational  awareness  even 
though  they  have  access  to  data  from  multiple  sensors.  Processing  the  data  into  actionable 
information  is  often  a  difficult  task.  One  aspect  of  this  task  is  determining  what  information 
should  be  reported.  Based  upon  subjective  opinions,  this  report  describes  what  information 
should  be  reported  to  a  Soldier  taking  fire  in  a  Forward  Operating  Base  (FOB),  Combat  Outpost 
(COP),  or  similar  area  and  how  to  quantify  its  accuracy.  While  the  metrics  were  developed  for 
acoustic  sensors,  they  can  also  be  applied  other  sensor  modalities. 

Acoustic  sensors  are  good  at  finding  the  direction  and  location  of  loud  acoustic  sources,  and 
fielded  systems  in  theater  currently  report  this  information  ( 1 ).  Requirements  for  detection  and 
false  alarm  rates  have  been  developed  for  several  acoustic  systems  (2,  3).  However,  a  literature 
search  did  not  find  requirements  or  guidelines  for  acoustic  classification  of  weapons  systems. 
Accurately  classifying  the  acoustic  signature  of  weapons  fire  is  a  difficult  problem  due  to 
atmospheric  attenuation,  dispersion,  multipath,  wind,  and  ambient  noise.  Although  many 
algorithms  have  been  developed  and  tested,  the  evaluation  criteria  remain  ill  defined.  The  criteria 
are  subjective  and  depend  upon  the  scenario  of  interest.  However,  by  developing  evaluation 
criteria,  we  provide  insight  into  what  information  is  important  and  provide  a  metric  for  scientists 
and  engineers  to  use  to  evaluate  their  algorithms. 

This  report  defines  evaluation  criteria  for  three  classification/discrimination  problems  based 
upon  the  judgments  of  two  military  analysts  with  experience  in  theater  and  an  acoustic  research 
scientist.  The  criteria  are  based  upon  scenarios  where  the  weapons  are  fired  without  suppressors 
from  an  open  area  and  the  rounds  are  landing  in  an  open  area.  The  classifier  receives  data  from 
an  acoustic  detection  and/or  bearing  estimation  system.  The  system  could  consist  of  a  single 
microphone,  an  array  of  microphones,  or  several  arrays  of  microphones.  The  classification 
algorithm  can  be  based  upon  the  combined  results  from  multiple  acoustic  sensors. 


2.  Classifier 


Classifier  performance  metrics  were  generated  to  distinguish  between  three  broad  categories  of 
events,  rather  than  to  identify  specific  weapon  types,  for  three  problems.  First,  the  classifier 
should  be  able  to  determine  whether  an  acoustic  event  occurred  at  the  point  of  origin  (POO)  or  at 
the  point  of  impact  (POI)  for  large-caliber  mortars,  rockets,  and  artillery.  For  events  occurring  at 
the  POO,  the  classifier  should  be  able  to  discriminate  between  direct-fire  and  indirect-fire 
weapons.  Lastly,  the  classifier  should  be  able  to  discriminate  between  acoustic  events  generated 
by  small  arms,  recoilless  weapons,  mortars,  and  artillery.  There  is  interest  in  further  classifying 
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each  weapons  system  category,  but  it  is  not  addressed  in  this  report.  The  classifier  should  only 
output  results  that  meet  the  specified  performance  criteria.  Partial  output  is  acceptable.  For 
example,  classifying  an  event  as  the  launch  of  a  direct-fire  weapon  of  unknown  type  is 
acceptable. 

The  metrics  are  described  in  the  form  of  a  confusion  matrix  with  probabilities  reported  instead  of 
numbers.  They  are  for  a  specific  range  where  the  acoustic  event  occurred  relative  to  the  closest 
point  on  the  perimeter  of  the  FOB  or  mobile  position.  Additionally,  the  performance  of  the 
classifier  at  smaller  ranges  should  be  equal  to  or  greater  than  the  performance  at  the  specified 
range.  Median  ambient  noise  levels  at  the  sensors  site  near  or  at  the  FOB  are  assumed  to  be 
60  dBA  or  less  without  filtering  and  will  typically  generate  a  low  false  alarm  detection  rate. 
However,  occasional  large  transient  acoustic  signals  generated  by  car  doors  slamming, 
helicopters  flying  by,  hammering,  yelling,  and  other  sources  of  acoustic  signals  will  potentially 
trigger  the  detection  algorithm.  These  events  are  considered  to  be  in  the  classifier  category  of 
noise. 

The  performance  criteria  for  discriminating  between  POO  and  POI  for  large  weapon  systems  and 
noise  are  shown  in  table  1.  Large-caliber  weapon  systems  include  mortars,  any  tube  artillery, 
and  rocket  artillery.  The  specified  range  for  POO  and  POI  is  2500  and  300  m,  respectively.  The 
table  indicates  that  the  probability  of  correctly  classifying  the  launch  or  impact  of  a  weapon  is 
75%  or  greater  and  the  probability  of  incorrectly  classifying  noise  as  a  launch  or  impact  of 
weapon  fires  is  25%  or  less. 

Table  1.  Probability  for  discriminating  between  launch  and  impact. 


Predicted  Weapon  Event 

Launch 

Impact 

Noise 

Actual  Weapon 
Event 

Launch 

>75  % 

<25  % 

<25  % 

Impact 

<25  % 

>75  % 

<25  % 

Noise 

<30% 

<30% 

>70% 

The  performance  metrics  for  discriminating  between  the  launch  of  direct-fire  weapons,  such  as  a 
small  arms  and  recoilless  rifle;  indirect-fire  weapons,  such  as  mortars  and  artillery;  and  noise  are 
shown  in  table  2.  The  specified  ranges  are  500  and  1500  m  for  direct-  and  indirect-fire  weapons, 
respectively.  The  table  indicates  that  the  probability  of  correctly  classifying  of  a  direct-fire 
weapon  is  75%  or  greater  and  the  probability  of  incorrectly  classifying  a  direct-fire  weapon  as  an 
indirect- fire  weapon  is  25%  or  less. 

Table  2.  Probability  for  discriminating  between  the  launch  of  direct-  and  indirect-fire  weapons. 


Predicted  Weapon  Type 

Direct 

Indirect 

Noise 

Actual  Weapon 
Type 

Direct 

>75  % 

<10% 

<25% 

Indirect 

<10% 

>75  % 

<25% 

Noise 

<30% 

<30% 

>70% 
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The  performance  metrics  to  discriminate  between  the  launch  of  small  arms,  recoilless  weapons, 
mortars,  and  artillery  are  shown  in  table  3.  The  specified  ranges  are  300,  500,  1500,  and  2500  m, 
respectively.  The  small  arms  category  includes  pistols,  carbines,  assault  rifles,  under-barrel 
grenade  launchers,  light  machine  guns,  general  purpose  machine  guns,  heavy  machine  guns 
(calibers  up  to  20  mm),  and  automatic  grenade  launchers.  The  recoilless  weapons  category 
includes  shoulder-fired  rockets  such  as  anti-tank  grenade  launchers  and  anti-tank  guided 
missiles.  The  table  indicates  that  the  probability  of  direct-fire  weapons  such  as  small  arms  being 
classified  as  indirect-fire  weapons  such  as  mortars  or  artillery  should  be  very  small:  5-10%. 

Table  3.  Probability  for  the  classification  of  the  launch  of  different  weapon  types. 


Small 

Arms 

Recoilless 

Mortar 

Artillery 

Noise 

Actual 

Weapon  type 

Small  arms 

>75  % 

<25  % 

<10% 

<5  % 

<25  % 

Recoilless 

<10% 

>75  % 

<25  % 

<5  % 

<25  % 

Mortar 

<10% 

<20% 

>75  % 

<25  % 

<25  % 

Artillery 

<  10% 

<10% 

<25  % 

>75  % 

<25  % 

Noise 

<10% 

<10% 

<10% 

<10% 

>70% 

3.  Conclusion 


Performance  metrics  for  classification  and  discrimination  of  acoustic  signal  from  weapons  fire 
were  developed  to  help  support  scientist  and  engineers  evaluate  their  algorithms.  We  developed 
performance  metrics  to  discriminate  between  broad  categories  of  events  including  a  noise 
category  rather  than  specific  weapon  systems.  It  is  anticipated  that  one  of  the  biggest  challenges 
for  meeting  the  performance  criteria  is  to  not  classify  noise  events  such  as  car  doors  slamming  as 
weapon  systems  fire.  Failure  to  meet  this  criterion  may  result  in  classification  results  in  future 
systems  being  ignored  or  future  systems  being  turned  off.  These  metrics  do  not  represent 
performance  criteria  for  any  system  under  development  and  are  not  associated  with  any 
Operational  Needs  Statement  (ONS)  from  any  theater  of  operations. 
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